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CLAIMS 

1 . An automated method of monitoring recognition accuracy in a voice 
recognition system comprising: 

retrieving voice recognition information produced by a voice recognition 
system in response to recognizing a user utterance, the voice recognition information 
comprising a recognized voice command associated with the user utterance and a 
reference to an audio file that includes the user utterance; 
playing the audio file; 

determining if the recognized voice command matches the user utterance 
included in the audio file; 

transcribing the user utterance to create a transcribed utterance, if the 
recognized voice command does not match the user utterance; and 

recording the transcribed utterance in association with the recognized voice 
command to monitor recognition accuracy of the voice recognition system. 

2. The method of Claim 1 , wherein the voice recognition information 
farther comprises information for a recognition grammar used to recognize the user 
utterance. 

3. The method of Claim 1, wherein the voice recognition information 
further comprises a confidence threshold associated with the recognized voice 
command. 

4. The method of Claim 1, wherein the voice recognition information 

25 further comprises a recognition delay associated with the recognized voice command. 

5. The method of Claim 1, wherein the transcribed utterance is recorded 
in a transcription log in association with the recognized voice command and the 
corresponding voice recognition information. 

30 

6. The method of Claim 5, wherein the voice recognition information 
includes the audio file that includes the user utterance. 
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7. The method of Claim 5, further comprising: 

analyzing information recorded in the transcription log to detect a type of 
recognition error. 

8. The method of Claim 7, further comprising: 

determining the source of the detected recognition error based on the type of 
recognition error. 

9. The method of Claim 8, further comprising: 
hypothesizing a solution to eliminate the source of error. 

10. The method of Claim 5, further comprising: 

determining recognition accuracy based on the information recorded in the 
transcription log. 

11. A system for monitoring recognition accuracy of a voice recognition 
system having at least one recognition grammar comprising: 

software for retrieving voice recognition information stored in a call log by a 
voice recognition system in response to recognizing a user utterance, the voice 
recognition information comprising a recognized voice command associated with the 
user utterance and a reference to an audio file that includes the user utterance; 

a transcription log produced by the transcription software, the transcription log 
comprising an entry associated with the user utterance, the entry comprising: 
the recognized voice command; and 

a transcribed utterance produced from transcribing the user utterance, 
if the user utterance included in the audio file does not match the recognized 
voice command associated with it; 

wherein the software determines a recognition accuracy for the voice 
recognition system by analyzing the entry recorded in the transcription log. 

12. The system of Claim 1 1 , wherein the voice recognition information 
further comprises information for a recognition grammar used to recognize the user 
utterance. 
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13. The system of Claim 11, wherein the voice recognition information 
further comprises a confidence threshold associated with the recognized voice 
command. 

14. The system of Claim 1 1 , wherein the voice recognition information 
further comprises: 

a recognition delay associated with the recognized voice command. 

1 5 . The system of Claim 1 1 , wherein the transcribed utterance is compared 
with the recognized voice command to determine if an error in recognition has 
occurred. 

16. The system of Claim 15, wherein the software records in an error log a 
type of recognition error associated with a recognized user utterance, if the 
transcribed utterance does not match the recognized voice command. 

1 7. The system of Claim 1 6, wherein the software records an in-grammar 
false reject (IGFR) type error, if the transcribed utterance is part of the recognition 
grammar but the user utterance is not. 

1 8. The system of Claim 16, wherein the software records an in-grammar 
false accept (IGF A) type error, if the transcribed utterance is part of the recognition 
grammar and the recognized voice command does not match the transcribed 
utterance. 

19. The system of Claim 16, wherein the software records an out-of- 
grammar false accept (OGFA) type error, if the transcribed utterance is not part of the 
recognition grammar and the recognized voice command is mistakenly matched with 
the transcribed utterance. 

20. The system of Claim 16, wherein the software records an out-of- 
grammar correct reject (OGCR) type error if the transcribed utterance is not part of 
the recognition grammar but the transcribed utterance is frequently recorded in the 
transcription log. 
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2 1 . The system of Claim 1 7, wherein: 

a source of error is determined to be a low confidence threshold, if an IGF A 
type error is detected. 

22. The system of Claim 1 7, wherein: 

a source of error is determined to be acoustic similarity between the 
recognized voice command and the user utterance, if an IGF A type error is detected. 

23 . The system of Claim 1 7, wherein the recognition grammar includes 
one or more acoustic models for a recognized voice command, wherein: 

a source of error is determined to be lack of sufficient acoustic models in the 
recognition grammar, if an IGFA type error is detected. 

24. The system of Claim 1 8, wherein a source of error is determined to be 
a high confidence threshold, if an IGFR type error is detected. 

25 . The system of Claim 1 8, wherein the recognition grammar includes 
one or more acoustic models for a recognized voice command, wherein: 

a source of error is determined to be lack of sufficient acoustic models in the 
recognition grammar, if an IGFR type error is detected. 

26. The system of Claim 1 8, wherein the voice recognition system is 
configured with a first recognition sensitivity threshold, wherein: 

a source of error is determined to be a low sensitivity threshold, if an IGFR 
type error is detected. 

27. The system of Claim 1 9, wherein: 

a source of error is determined to be acoustic similarity between the 
recognized voice command and the user utterance, if an OGFA type error is detected. 

28. The system of Claim 19, wherein the voice recognition system is 
configured with a first recognition sensitivity threshold, wherein: 
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a source of error is determined to be a high sensitivity threshold, if an OGFA 
type error is detected. 

29. The system of Claim 19, wherein a source of error is determined to be 
5 a low confidence threshold, if an OGFA type error is detected. 

30. The system of Claim 20, wherein the recognition grammar includes 
one or more acoustic models for the recognized voice command, wherein: 

a source of error is determined to be lack of sufficient acoustic models in the 
recognition grammar, if an OGCR type error is detected. 

31. A method for automatically improving a voice recognition system 
comprising: 

determining voice recognition accuracy by analyzing voice recognition 
information produced by a voice recognition system in response to receiving a user 
utterance, the voice recognition system comprising a recognition grammar and a first 
configuration, the voice recognition information comprising a recognized voice 
command associated with the user utterance; 

detecting a recognition error, if the recognized voice command does not match 
a transcribed utterance for the user utterance; 

determining at least one source of error for the detected recognition error; 
hypothesizing at least one solution to eliminate the source of error; and 
modifying the first configuration based on the hypothesized solution to 
implement a second configuration for the voice recognition system with better 
recognition accuracy. 

32. The method of Claim 3 1 , further comprising: 
testing the voice recognition system to detect improvements in voice 

recognition accuracy; and 
30 restoring the first configuration, if voice recognition accuracy is not improved. 

33. The method of Claim 32, wherein the voice recognition information 
further comprises information for a recognition grammar used to recognize the user 
utterance. 
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34. The method of Claim 33, wherein the voice recognition information 
further comprises a confidence threshold associated with the recognized voice 
command. 

35. The method of Claim 34, wherein the voice recognition information 
further comprises a recognition delay associated with the recognized voice command. 

36. The method of Claim 35, wherein an in- grammar false accept (IGF A) 
type error is detected, if the transcribed utterance is part of the recognition grammar 
and the recognized voice command does not match the transcribed utterance. 

37. The method of Claim 35, wherein an in-grammar false reject (IGFR) 
type error is detected, if the transcribed utterance is part of the recognition grammar 
but the user utterance is not. 

38. The method of Claim 35, wherein an out-of- grammar false accept 
(OGFA) type error is detected, if the transcribed utterance is not part of the 
recognition grammar and the recognized voice command is mistakenly matched with 
the transcribed utterance. 

39. The method of Claim 35, wherein an out-of-grammar correct reject 
(OGCR) type error is detected, if the transcribed utterance is not part of the 
recognition grammar but the transcribed utterance is frequently recorded in the 
transcription log. 

40. The method of Claim 36, wherein a source of error is determined to be 
a low confidence threshold, if an IGFA type error is detected. 

41 . The method of Claim 36, wherein a source of error is determined to be 
acoustic similarity between the recognized voice command and the user utterance, if 
an IGFA type error is detected. 
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42. The method of Claim 36, wherein the recognition grammar includes 
one or more acoustic models for a recognized voice command, wherein a source of 
error is determined to be lack of sufficient acoustic models in the recognition 
grammar, if an IGFA type error is detected. 

43. The method of Claim 37, wherein a source of error is determined to be 
a high confidence threshold, if an IGFR type error is detected. 

44. The method of Claim 37, wherein the recognition grammar includes 
one or more acoustic models for a recognized voice command, wherein a source of 
error is determined to be lack of sufficient acoustic models in the recognition 
grammar, if an IGFR type error is detected. 

45. The method of Claim 37, wherein the voice recognition system is 
configured with a first recognition sensitivity threshold, wherein a source of error is 
determined to be a low sensitivity threshold, if an IGFR type error is detected. 

46. The method of Claim 38, wherein a source of error is determined to be 
acoustic similarity between the recognized voice command and the user utterance, if 
an OGFA type error is detected. 

47. The method of Claim 38, wherein the voice recognition system is 
configured with a first recognition sensitivity threshold, wherein a source of error is 
determined to be a high sensitivity threshold, if an OGFA type error is detected. 

48. The method of Claim 38, wherein a source of error is determined to be 
a low confidence threshold, if an OGFA type error is detected. 

49. The method of Claim 39, wherein the recognition grammar includes 
one or more acoustic models for a recognized voice command, wherein a source of 
error is determined to be lack of sufficient acoustic models in the recognition 
grammar, if an OGFR type error is detected. 
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50. A method of improving recognition accuracy and efficiency in a voice 
recognition system having a grammar, wherein a received utterance is accepted if the 
utterance is an acoustic match with at least one term included in the grammar and 
rejected otherwise, the method comprising: 

5 determining if the utterance has been falsely accepted by the voice recognition 

system; and 

removing a term from the grammar, if the utterance is part of the grammar but 
is falsely accepted as another term included in the grammar due to acoustic similarity. 

5 1 . The method of Claim 50, further comprising: 
adding to the grammar one or more synonyms which are acoustically 

dissimilar to the utterance if it has been falsely accepted. 

52. The method of Claim 50, wherein the voice recognition system 
comprises at least one acoustic model for the utterance, the method further 
comprising: 

adding to the voice recognition system an additional acoustic model for the 
utterance, if the utterance is part of the grammar, but is falsely rejected due to lack of 
sufficient alternative acoustic models. 

53. The method of Claim 52, wherein the additional acoustic model is one 
that is associated with the rejected utterance. 

54. The method of Claim 50, wherein the voice recognition system has a 
25 confidence threshold for adjusting recognition accuracy based on degree of acoustic 

similarity between the utterance and one or more acoustic models included in the 
voice recognition system, wherein the recognition accuracy is improved if the 
confidence threshold is higher, the method further comprising: 

lowering the confidence threshold, if the utterance is part of the grammar, but 
30 is falsely rejected due to a high confidence threshold. 

55. The method of Claim 50, the voice recognition system having a 
sensitivity threshold for accepting utterances, wherein a higher sensitivity allows for a 
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larger number of utterances to be considered by the voice recognition system, the 
method further comprising: 

lowering the sensitivity threshold if a predetermined amount of noise is 
undesirably considered by the voice recognition system. 

56. The method of Claim 55, further comprising: 

increasing the sensitivity threshold, if the utterance is part of the grammar, but 
is falsely rejected due to a low sensitivity threshold. 

57. The method of Claim 50, the voice recognition system having a 
timeout threshold and an end-of-speech threshold for accepting an utterance having a 
certain audio length or audio delay, the method further comprising: 

adjusting the timeout and the end-of-speech thresholds, if a predetermined 
amount of noise is undesirably considered for acceptance by the system. 

58. The method of Claim 50, wherein the grammar has a perplexity 
associated with a number of decision levels and terms included in the grammar, the 
method further comprising: 

reducing grammar perplexity, if recognition of the utterance is delayed beyond 
a threshold value. 

59. The method of Claim 50, further comprising: 

monitoring frequently received utterances that are rejected because they are 
not part of the grammar; and 
25 adding the frequently received utterances to the grammar. 

60. A method performed on a computer for monitoring recognition 
accuracy in a voice recognition system comprising: 

recording a user utterance in an audio file; 
30 storing a voice command which was recognized by the voice recognition 

system for the user utterance; 

presenting the audio file and the voice command to an operator for selection; 
playing the user utterance from the audio file for the operator upon selection 
by the operator; 
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receiving a transcription of the user utterance; and 

associating the transcription with the recognized voice command. 
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